Data Exploration on Kaggle's House Prices Dataset¶
by Houssem Menhour¶
Preliminary Wrangling¶
This is a popular dataset from Kaggle with information that allows for predicting sales prices and practicing feature engineering, RFs, and gradient boosting. In this project, I'll be focusing on data exploration and visualization.
The full data description is available here.
In [115]:
# import all packages and set plots to be embedded inline
import numpy as np
import pandas as pd
pd.set_option('display.max_columns', None)
# Plotting
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set_style("whitegrid")
sns.set()
Load in your dataset and describe its properties through the questions below. Try and motivate your exploration goals through this section.
In [116]:
data_path = "../datasets/P5/train.csv" #using the training dataset
df = pd.read_csv(data_path)
df.info()
In [117]:
df.describe()
Out[117]:
Notes
- With so many columns, it is difficult to explore the data visually, let's get a list of numerical values and another for categorical ones
- Some numerical columns actually represent categorical information (
MSSubClass), we should fix that early on - Columns with categorical variables should be converted from
objecttocategory - From a quick look, we can see several columns with a lot of
nullvalues, let's start by vizualizing them for better understanding, then getting rid of them. Also, we need to populate those with lessnullvalues - Since the dataset was meant for predicting
SalePriceusing regression, let's see how each individual variable correlate to it
In [118]:
#Get a list of all numerical and categorical features
def cat_num_split(df_, y_inc = False):
"""Get seperate lists for numerical and categorical data"""
numerical_list = []
categorical_list = []
tmp = None
if y_inc:
tmp = df_.drop(columns=["SalePrice"]).columns.tolist()
else:
tmp = df_.columns.tolist()
for col in tmp:
if (df_[col].dtype == 'object' or str(df_[col].dtype) == 'category'):
categorical_list.append(col)
else:
numerical_list.append(col)
return numerical_list, categorical_list
num_list, cat_list = cat_num_split(df)
In [119]:
df[num_list]
Out[119]:
In [120]:
df[cat_list]
Out[120]:
In [121]:
#Create a copy of the dataset before any modifications
df2 = df.copy()
#Map every class number to CS{number} format
#df2['MSSubClass'] = df2['MSSubClass'].map('CS{}'.format)
df2['MSSubClass'] = df2['MSSubClass'].map(lambda x: 'CS{}'.format(x))
df2['MSSubClass']
#Recompile num/cat lists
num_list, cat_list = cat_num_split(df2)
What is the structure of your dataset?¶
- This dataset comes in 1460 entries and 81 columns, 38 of which are numerical while 43 are categorical.
- Some of the categorical columns represent ordinal values, this mainly applies to variables of a quality related value
What is/are the main feature(s) of interest in your dataset?¶
- I'm interested in finding the best features to predict
SalePriceof a given house.
What features in the dataset do you think will help support your investigation into your feature(s) of interest?¶
- I expect features related to the area of the land, number of floors, and quality/condition of the house to be the most effective. Since there are many in this dataset, I'll leave listing them for later along the way.
Univariate Exploration¶
In this section, investigate distributions of individual variables. If you see unusual points or outliers, take a deeper look to clean things up and prepare yourself to look at relationships between variables.
Find and fix missing data¶
In [122]:
def label_null_mapping(df_):
label_nas = df_.isnull().sum().astype('bool')
label_nas = label_nas.index.map(lambda x: x if label_nas[x] else "")
return label_nas
label_nas = label_null_mapping(df2)
In [123]:
#Display missing values
plt.figure(figsize=(12,8))
plt.suptitle('Missing Values in the Dataset')
sns.heatmap(df2.isnull(), yticklabels=False, xticklabels=label_nas ,cbar = False, cmap='viridis')
plt.show()
- After calculating the number of missing values in each column and displaying them in a way that highlets them, we can clearly see that some features can't be relied on for any meaningfull analysis, I will remove them in the next step.
- Features with less missing data will be filled with either the mean/mode value of the column for numerical/categorical features respectively.
In [124]:
#Remove columns with too many missing values
null_values_per_col = np.sum(df2.drop(["SalePrice"], axis=1).isnull(), axis=0)
max_na = int(df2.shape[0]/5.0) #allowing up to 1/5 (20%) of the data to be missing
cols_to_remove = []
for col in df2.drop(["SalePrice"],axis=1).columns.tolist():
if null_values_per_col[col] > max_na:
cols_to_remove.append(col)
df2.drop(col, axis=1, inplace=True)
print("New shape of the training set is: (%d,%d)" %df2.shape)
print("The removed columns are: " + str(cols_to_remove))
In [125]:
#Display again
label_nas = label_null_mapping(df2)
plt.figure(figsize=(12,8))
plt.suptitle('Missing Values in the Dataset')
sns.heatmap(df2.isnull(), yticklabels=False, xticklabels=label_nas ,cbar = False, cmap='viridis')
plt.show()
In [126]:
col_interest = [x for x in label_nas if x != ""]
print(col_interest)
In [127]:
# Fill the rest of null values
# With median for numerical, and mode for categorical
df2.fillna(df2.mean(), inplace = True)
for col in col_interest:
df2[col].fillna(df2[col].mode()[0], inplace = True)
Reorganize columns¶
To better represent categorical data, especially when ordinal
In [128]:
#remap some columns to have numerical data
rep_map_1 = ["NA", "Po", "Fa", "TA", "Gd", "Ex"]
rep_map_2 = ["NA", "No", "Mn", "Av", "Gd"]
rep_map_3 = ["NA", "Unf", "LwQ", "Rec", "BLQ", "ALQ", "GLQ"]
rep_map_4 = ["NA", "MnWw", "GdWo", "MnPrv", "GdPrv"]
rep_map_5 = ["N", "P", "Y"]
rep_map_6 = ["NA", "Unf", "RFn", "Fin"]
rep_map_7 = np.linspace(1,10, 10).astype("int").astype("str").tolist()
rep_map_8 = ["Sev", "Mod", "Gtl"]
rep_map_9 = ["ELO", "NoSeWa", "NoSewr", "AllPub"]
rep_map_10 = ["IR3", "IR2", "IR1", "Reg"]
ordinal_var_dict = {"ExterCond": rep_map_1,
"ExterQual": rep_map_1,
"BsmtCond": rep_map_1,
"BsmtQual": rep_map_1,
"HeatingQC": rep_map_1,
"KitchenQual": rep_map_1,
"GarageCond": rep_map_1,
"BsmtExposure": rep_map_2,
"BsmtFinType1": rep_map_3,
"BsmtFinType2": rep_map_3,
"PavedDrive": rep_map_5,
"GarageFinish": rep_map_6,
#"OverallCond": rep_map_7,
#"OverallQual": rep_map_7,
"LandSlope": rep_map_8,
"Utilities": rep_map_9,
"LotShape": rep_map_10,
}
cols_ordinal = ["ExterCond",
"ExterQual",
"BsmtCond",
"BsmtQual",
"HeatingQC",
"KitchenQual",
"GarageCond",
"BsmtExposure",
"BsmtFinType1",
"BsmtFinType2",
"PavedDrive",
"GarageFinish",
"LandSlope",
"Utilities",
"LotShape"]
In [129]:
num_list, cat_list = cat_num_split(df2)
# Reencode ordinal categories
for var in ordinal_var_dict:
ordered_var = pd.api.types.CategoricalDtype(ordered = True,
categories = ordinal_var_dict[var])
df2[var] = df2[var].astype(ordered_var)
# Reencode the rest
for col in cat_list:
if col not in ordinal_var_dict:
df[col] = df[col].astype('category')
- I created a mapping scheme for ordinal categories and kept a list of features meeting this criteria for later use.
- applying the mapping and data type conversion gives us the following DataFrame:
In [130]:
df2.info()
In [131]:
num_list, cat_list = cat_num_split(df2)
- Using some loops and pairplot features, we can obtain enough graphics to better understand the data and the distribution of each of its features.
Numerical Features¶
In [132]:
f = plt.figure(figsize=(14, 48))
gs = f.add_gridspec(len(num_list), 4)
for i, col in enumerate(num_list):
ix, iy= int(i/2), (2 if i%2 else 0)
ax = f.add_subplot(gs[ix,iy])
ax = sns.boxplot(df2[col]);
ax = f.add_subplot(gs[ix,iy+1])
ax = sns.distplot(df2[col], kde=False);
f.tight_layout()
Categorical Features (Ordinal)¶
In [133]:
cols_interest = [x for x in cat_list if x in cols_ordinal]
f = plt.figure(figsize=(20, 50))
gs = f.add_gridspec(len(cols_interest), 4)
for i, col in enumerate(cols_interest):
ax = f.add_subplot(gs[int(i/4),i%4])
#sns.boxplot(df2[col], kind="box");
ax = sns.countplot(df2[col])
ax.set_xticklabels(ax.get_xticklabels(), ha = 'right', rotation=60)
#sns.distplot(df2[col], kde=False);
# add annotations
n_points = df.shape[0]
col_counts = df2[col].value_counts()
locs, labels = plt.xticks() # get the current tick locations and labels
# loop through each pair of locations and labels
for loc, label in zip(locs, labels):
# get the text property for the label to get the correct count
count = col_counts[label.get_text()]
pct_string = '{:0.1f}%'.format(100*count/n_points)
# print the annotation just below the top of the bar
plt.text(loc, count+10, pct_string, ha = 'left', color = 'b', rotation=60, fontsize=10)
f.tight_layout()
Categorical Features (Nominal)¶
In [134]:
cols_interest = [x for x in cat_list if not (x in cols_ordinal)]
f = plt.figure(figsize=(25, 100))
gs = f.add_gridspec(len(cols_interest), 5)
for i, col in enumerate(cols_interest):
ax = f.add_subplot(gs[int(i/5),i%5])
#sns.boxplot(df2[col], kind="box");
ax = sns.countplot(df2[col])
ax.set_xticklabels(ax.get_xticklabels(), ha = 'right', rotation=60)
#sns.distplot(df2[col], kde=False);
# add annotations
n_points = df.shape[0]
col_counts = df2[col].value_counts()
locs, labels = plt.xticks() # get the current tick locations and labels
# loop through each pair of locations and labels
for loc, label in zip(locs, labels):
# get the text property for the label to get the correct count
count = col_counts[label.get_text()]
pct_string = '{:0.1f}%'.format(100*count/n_points)
# print the annotation just below the top of the bar
plt.text(loc, count+10, pct_string, ha = 'left', color = 'b', rotation=60, fontsize=10)
f.tight_layout()
- I made 3 grids for better readability, one for each of numerical features, ordinal categories and nominal categories
- Many variables in both numerical and categorical form are severly unbalanced, with ~95% of the values falling in one category or value and the rest being mostly outliers, These won't help us draw any conclusions from their effect on the
SalePriceand we will get rid of them in the next step
In [135]:
num_to_drop = ['Id', 'LowQualFinSF', '3SsnPorch', 'PoolArea', 'BsmtFinSF2', 'BsmtHalfBath', 'KitchenAbvGr', 'EnclosedPorch', 'ScreenPorch', 'MiscVal']
cat_to_drop = ['Street', 'Utilities', 'LandSlope', 'Condition2', 'RoofMatl', 'Heating', 'GarageQual', 'GarageCond']
df2 = df2.drop(num_to_drop + cat_to_drop, axis=1)
Skewness Analysis and Normalization¶
MoSoldis a periodic value, where 12 should be interpreted as closer to 1 than to 10. That's why it is better to apply a trigonometric transform to it if we try to fit a model to this data but that's beyond the scope of the project so I will skip it.SalePriceand other numerical features are very skewed, will need a log transform to make them as close to a normal distribution as possible. I will start by visualising that aspect inSalePricethen proceed to fix it as well as the other features, and replot everything at the end.
In [136]:
#Check the skewness of SalePrice
skewness = df2['SalePrice'].skew()
f, ax = plt.subplots(figsize=(10, 5))
sns.distplot(df2['SalePrice'], color="b", kde=True);
ax.set(ylabel="Frequency")
ax.set(xlabel="SalePrice")
ax.set(title="SalePrice distribution (skew = {})".format(round(skewness,2)))
plt.show()
In [137]:
#Apply log1p to reduce skewness
skewness = np.log1p(df2['SalePrice']).skew()
f, ax = plt.subplots(figsize=(10, 5))
sns.distplot(np.log1p(df2['SalePrice']), color="b");
ax.set(ylabel="Frequency")
ax.set(xlabel="SalePrice")
ax.set(title="SalePrice distribution (skew = {})".format(round(skewness,2)))
plt.show()
- We can see a significant improvement to the distribution of
SalePrice, we will apply the same transform to all skewed numerical data
In [138]:
skewness1 = df2['SalePrice'].skew()
skewness2 = np.log1p(df2['SalePrice']).skew()
f = plt.figure(figsize=(15, 5))
gs = f.add_gridspec(1, 2)
ax = f.add_subplot(gs[0,0])
sns.distplot(df2['SalePrice'], color="b", kde=True);
ax.set(title="(skew = {})".format(round(skewness1,2)))
#ax.set(ylabel="Frequency")
#ax.set(xlabel="SalePrice")
ax = f.add_subplot(gs[0,1])
ax = sns.distplot(np.log1p(df2['SalePrice']), color="b");
ax.set(title="(skew = {})".format(round(skewness2,2)))
f.suptitle('Sale Price distribution', fontsize=16)
f.tight_layout()
In [139]:
num_list, cat_list = cat_num_split(df2)
# Log transform of the skewed numerical features to lessen impact of outliers
skewness = df2[num_list].skew()
skewness = skewness[abs(skewness) > 0.62] #Less than this and it will transform YearBuilt
print(str(skewness.shape[0]) + " skewed numerical features to log transform")
skewed_features = skewness.index
print(skewed_features)
df2[skewed_features] = np.log1p(df2[skewed_features])
In [26]:
f = plt.figure(figsize=(14, 48))
gs = f.add_gridspec(len(num_list), 4)
for i, col in enumerate(num_list):
ix, iy= int(i/2), (2 if i%2 else 0)
ax = f.add_subplot(gs[ix,iy])
ax = sns.boxplot(df2[col]);
ax = f.add_subplot(gs[ix,iy+1])
ax = sns.distplot(df2[col], kde=False);
f.tight_layout()
- This grid shows that the trasform successfully improved the distribution of the data
Discuss the distribution(s) of your variable(s) of interest. Were there any unusual points? Did you need to perform any transformations?¶
- Many of the variables in this dataset had skewed distributions, the most sever of which were simply removed, while the rest can be normalized.
Of the features you investigated, were there any unusual distributions? Did you perform any operations on the data to tidy, adjust, or change the form of the data? If so, why did you do this?¶
SalePriceand several other variables were skewed, I need to normalize them by applyingnp.log1p.MoSalecould also be transformed to better show its periodic nature, but that's not needed for this project.- The sever ones also happened to be the ones with the highest numbers of outliers.
Bivariate Exploration¶
In this section, investigate relationships between pairs of variables in your data. Make sure the variables that you cover here have been introduced in some fashion in the previous section (univariate exploration).
Finding Correlations¶
In [27]:
#Correlation matrix
corr_mat = df2.corr().abs()
# Find most important features relative to target
corr_mat.sort_values(["SalePrice"], ascending = False, inplace = True)
print(corr_mat.SalePrice)
cols_top_corr = corr_mat.SalePrice.index[1:13].tolist()
- The previous few lines of code allow us to find the features that are more correlated to the variable in question
SalePrice. - We can do better and find correlation between all numerical variables, and reduce depenency between them accordingly. While this is a Multivariable Exploration, I find it important enough to do early on.
In [28]:
#Correlation matrix between features
corr_mat = df2.drop(columns=["SalePrice"]).corr().abs()
In [29]:
#Plot the heatmap with seaborn
plt.figure(figsize=(15,15))
sns.heatmap(corr_mat,
xticklabels=corr_mat.columns,
yticklabels=corr_mat.columns)
Out[29]:
- In this heatmap, we can spot a few correlated variables, this kind of relations is undesired if we want to feed the data to a regression model, so let's fix that by removing the most noticeable ones.
In [30]:
#List the highly correlated columns
corr_tmp = corr_mat.unstack()
corr_tmp = corr_tmp.sort_values(kind="quicksort", ascending = True)
print(corr_tmp[-len(corr_mat)-20:-len(corr_mat)])
In [31]:
corr_to_drop = ["GarageYrBlt", "GarageCars", "GrLivArea", "TotalBsmtSF"]
cols_top_corr = [x for x in cols_top_corr if not (x in corr_to_drop)]
df2 = df2.drop(corr_to_drop, axis=1)
SalePrice vs Categorical variables¶
In [32]:
num_list, cat_list = cat_num_split(df2)
f = plt.figure(figsize=(25, 100))
gs = f.add_gridspec(len(cat_list), 5)
for i, col in enumerate(cat_list):
ax = f.add_subplot(gs[int(i/5),i%5])
#sns.boxplot(df2[col], kind="box");
ax = sns.boxplot(x=col, y="SalePrice", data=df2)
#sns.distplot(df2[col], kde=False);
ax.set_xticklabels(ax.get_xticklabels(), ha = 'right', rotation=60)
f.tight_layout()
SalePrice vs Numerical variables¶
In [33]:
g = sns.pairplot(df2[num_list], size=4, kind="reg");
In [34]:
g = sns.pairplot(df2[cols_top_corr + ['SalePrice']], size=4, kind="reg");
Talk about some of the relationships you observed in this part of the investigation. How did the feature(s) of interest vary with other features in the dataset?¶
- With a quick glance at the features, we can see that not all of them show a clear effect on our variable of interest
SalePrice. - Among the most correlated we find the Lot Area, Overall Quality, and Year Built variabls, 1st Floor SquareFeet seems to contribute significantly as well.
- Surprisingly, 2nd Floor SquareFeet doesn't show the same correlation as 1st Floor's, maybe due to having less data on that.
- The time of selling doesn't seem to affect the price in a significant way, in terms of both monthly and yearly.
- Less common and niche technical details such as the Basement Unfinished SF, WoodDeck SF and OpenPrch SF showed little to none correlation as well, which is to be expected considering how few of the entries actually had them.
- Among categorical variables, some stand out as highly correlated to
SalePrice, mostly Quality related, and include Basment, Kitchen, Heating, Drive Way ... - The Lost Shape surprisingly has a negative correlation with
SalePrice, with the most irregular shaped ones fetching the highest prices.
Did you observe any interesting relationships between the other features (not the main feature(s) of interest)?¶
1stFlrSFandGarageAreashow a very high correlation.- Same thing goes for
YearBuiltandGarageArea, year over year, Garage areas has been increasing, this is the opposite of my expectations since I thought the increasing demand and prices of land will push the garage area to a lower average. - We can also see a clear correlation between
YearBuiltand Overall Quality variable, not surprising considering the higher standards and more advanced technologies.
Multivariate Exploration¶
Create plots of three or more variables to investigate your data even further. Make sure that your investigations are justified, and follow from your work in the previous sections.
In [35]:
cols_ordinal = [x for x in df2.columns.tolist() if x in cols_ordinal] # or x in cols_top_corr)]
cols_nominal = [x for x in df2.columns.tolist() if not (x in cols_ordinal)]
cols_interest.append('SalePrice')
cols_stripplot = ['OverallQual', 'FullBath', 'TotRmsAbvGrd', 'Fireplaces']
In [36]:
f = plt.figure(figsize=(100, 100))
gs = f.add_gridspec(len(cols_top_corr), len(cols_ordinal))
for i, col in enumerate(cols_top_corr):
for j, col2 in enumerate(cols_ordinal):
ax = f.add_subplot(gs[i,j])
#sns.boxplot(df2[col], kind="box");
#{x, y}_jitter does nothing iwth scatterplot according to docs
if col in cols_stripplot:
ax = sns.stripplot(x=col, y="SalePrice", hue=col2, data=df2, alpha = 0.3, jitter = 0.2)
else:
ax = sns.scatterplot(x=col, y="SalePrice", hue=col2, data=df2, alpha = 0.3)
#sns.distplot(df2[col], kde=False);
#ax.set_xticklabels(ax.get_xticklabels(), ha = 'right', rotation=60)
f.tight_layout()
